Unsupervised Models of Images by Spikeand-Slab RBMs

The spike and slab Restricted Boltzmann Machine (RBM) is defined by having both a real valued “slab” variable and a binary “spike” variable associated with each unit in the hidden layer. In this paper we generalize and extend the spike and slab RBM to include non-zero means of the conditional distribution over the observed variables given the binary spike variables. We also introduce a term, quadratic in the observed data that we exploit to guarantee that all conditionals associated with the model are well defined – a guarantee that was absent in the original spike and slab RBM. The inclusion of these generalizations improves the performance of the spike and slab RBM as a feature learner and achieves competitive performance on the CIFAR-10 image classification task. The spike and slab model, when trained in a convolutional configuration, can generate sensible samples that demonstrate that the model has captured the broad statistical structure of natural images.